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ABSTRACT 

* In the fall of 1995 the National Center for Education 
Statistics (NCES) held a conference to stimulate dialogue about 
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methodology, and technology and the implications of these 
developments for the nation's education statistics program. This 
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study the NCES undertook to clarify its r es pons i b i 1 i t i es over the 
next decade. An introductory section provides a synopsis of the major 
themes and conclusions from the conference and its commissioned 
papers. The second section describes the current foundation of the 
NCES, outlining its core functions, operat ing . pr inc ipl es , and 
program. The third section summarizes some new directions that the 
NCES could pursue to provide information for policy, research , and 
practice in American education and addresses some important 
methodological and technological opportunities. A concluding section 
affirms that the most fundamental change the NCES must make is to 
emerge as a provider of information services and systems rather than 
a primary collector and provider of data per se. Data collection that 
occurs independently of front“line admini s t ra t i ve and teaching 
systems is likely to diminish in the future. An outline of the 
conference agenda is presented. (Contains five graphs and six 
references.) (SLD) 
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Qommissioner’s Statement 

In the fall of 1995, the National Center for Education Statistics (NCES) held a 
conference to stimulate dialogue about future developments in the fields of 
education, statistical methodology, and technology, as well as to explore the 
implications of such developments for the nation's education statistics pro- 
gram. This "Futures Conference" was unique for NCES because it attempted 
to combine considerations in all of these fields in order to stimulate the cross- 
fertilization and generation of ideas that might not emerge when discussing 
the topics separately. At this conference, the authors presented commissioned 
papers on targeted issues that were expected to be important over the next few 
years, and the discussants provided their comments. 

From several perspectives, I believe the conference was highly successful. First, 
staff from NCES actively participated in all of the deliberations. As a result, 
they became personally engaged in the process of considering alternative 
futures for their agency. Since the "corporate culture" of this agency is to solic- 
it and build on staff creativity, their participation and interest in this confer- 
ence was vital. Second, both the formal and informal discussions generated 
many new ideas. The conference, as such, accomplished far more than the col- 
lection of commissioned papers alone could have because of the active inter- 
play of ideas. Finally, many stakeholders in NCES's future saw this conference 
as a clear signal of the agency's commitment to continued improvement of the 
usefulness and quality of our surveys and data products. The stakeholders' 
positive response to the meeting was further reinforced by their expressions of 
interest in continuing to help in important ways. The success of the conference 
lies not in the sum of the individual presentations, rather in an overall per- 
spective that provides guidance toward the future. 

This document and its companion publication. Conference Proceedings, will 
serve as a concrete reference to ensure that the stimulating ideas exchanged at 
the Futures Conference are not forgotten. While the quality of the discussion at 
the meeting was exceptional, one cannot expect to absorb everything said dur- 
ing a two-day conference. Thus, it is important to have a record that the par- 
ticipants can refer to this year, next year, or five years from now. Moreover, 
this publication will provide a way to share those ideas with others who could 
not participate in the conference. For instance, NCES has many customers and 
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other stakeholders who have expressed keen interest in the conference proceed- 
ings and whose advice and considerations are welcome as a means to sustain the 
dialogue about NCES's future. 

It is clear that if NCES wants to continue as a key player in providing information 
for education policy and decision making to the American public, policymakers, 
education researchers, and educators nationwide, it must continually reevaluate 
its program and products. In the future, we expect that NCES will receive 
requests for more of the kinds of products and services that it already provides. 
Also, we expect demands for new perspectives — on covering new topical areas, 
implementing new technologies, and adopting new methodologies. Already, 
major recent changes in the field of education are shaping our future program — 
for example, widespread innovations to achieve education reform, efforts to 
adopt both curriculum and performance standards, and examination of educa- 
tion in the United States within an international context. Not only are method- 
ological advances creating opportunities to produce statistics in ways that may be 
more efficient and effective, but also technological developments are changing 
the world in which we create data and disseminate our products more rapidly 
than ever before. The Futures Conference and this publication provide a new 
vision for NCES — a vision that acknowledges the constraints on the resources of 
governmental agencies at the end of the 20th century, as well as clearly empha- 
sizes the opportunities that can be achieved with innovative methodologies and 
technologies and through close attention to the priorities for statistical knowledge 
in the field of education. 

This contribution to envisioning NCES's future is occurring at a pivotal time of 
transition. The Futures Project was conceptualized under the leadership of the 
first Commissioner of Education Statistics, Emerson J. Elliott and carried through 
under the stewardship of Jeanne E. Griffith as Acting Commissioner. I plan to use 
this publication in the upcoming years as a source of ideas for planning and 
thinking and as a foundation for long-term change in the organization. 



Pascal D. Forgione, Jr. 
Commissioner 
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Introduction 



In 1995, approximately 65 million Americans participated in elementary, sec- 
ondary, or postsecondary education in the United States (U.S. Department of 
Education 1995b). Young people spent from one-fourth to one-half of their wak- 
ing hours in school and school-related activities, while Americans of all ages con- 
tinued to pursue some form of active learning that added to their repertoire of 
knowledge and skills. To serve these students, the nation's schools, colleges, and 
universities directly employed some 11.2 million people, even more than in the 
health industry. In addition, these direct services supported a substantial number 
of additional jobs in companies serving education through the production of 
everything from buses and computers to textbooks and software. All told, the 
nation spent more than $500 billion^ on formal education, or approximately 7.4 
percent of Gross Domestic Product (U.S. Department of Commerce 1995). 
Probably no other single activity has occupied so prominent a place in family, 
community, and working life. 

The primary purpose of the 
National Center for Education 
Statistics (NCES) is to describe 
this education enterprise and 
inform the nation about it. 

Congress charges NCES with col- 
lecting and reporting "statistics 
and information showing the con- 
dition and progress of education 
in the United States and other 
nations in order to promote and 
accelerate the improvement of 
American education."- In doing 
so, the Center conducts a range of 
ongoing national surveys examin- 
ing early childhood education, 
elementary and secondary educa- 
tion, postsecondary education, 
adult literacy, and the nation's 
libraries. Further, in cooperation with many other countries, it supports internation- 
al surveys that aid in comparing educational progress and processes across nations. 
NCES carries out numerous analyses of these data and annually prepares more than 



Percentage of the Population Enrolled in 
School, BY Age, October 1993 




SOURCE: U.S. Department of Education, National Center for Education 
Statistics, The Condition of Education 1995, Washington, D.C., 26-27. 
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100 reports targeted toward policymakers, educa- 
tors, researchers, and the American people. 

Doing this job well would be necessary no matter 
what the focus, but when the subject assumes the 
magnitude and importance of education, this 
responsibility takes on special significance. 
Therefore, NCES must ensure that it continues to 
describe education fully and accurately and that it 
performs this function efficiently and thoroughly — 
in other words, that it remains well informed about 
important education issues, key advances in meth- 
ods, and new developments in the technology of collecting, managing, analyzing, 
and reporting large amounts of information. 

To this end, NCES undertook an in-depth examination of how best to direct its 
responsibilities for collecting and reporting information on education over the next 
decade. Three principal questions guided this effort; 

1) What are the major issues and trends in education that NCES should aim 
to address through the first decade of the next century? 

2) What are the most important advances in methods for collecting and ana- 
lyzing information that should guide how NCES surveys are designed 
and used? 

3) What opportunities do technological advances in data management and 
communications present for improving data collection and analysis and 
for disseminating findings and information effectively? 

To help answer these questions, NCES conducted four activities: 1) a survey of lead- 
ing educators and researchers, asking them to answer one or more of the three 
questions listed above; 2) commissioned papers addressing key topics suggested by 
the survey results; 3) a conference where the authors of the commissioned papers 
presented their work, with subsequent discussion by NCES staff and external 
reviewers; and 4) a published volume. From Data to Information: New Directions for the 
National Center for Education Statistics, of the commissioned papers and discussants' 
comments. This paper summarizes and synthesizes the results of this work and 
consists of four major sections. This first introductory section provides a synopsis of 
the major themes and conclusions emerging from the papers and the conference. 
The second section describes the current foundation of NCES, delineating its core 
functions, operating principles, and program of work. Tlie third summarizes some 
new directions that NCES could pursue to provide information for policy, research. 
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and practice in American education, and also addresses some important method- 
ological and technological opportunities. The paper ends with a brief conclusion. 

Two dominant themes emerged from this collaborative effort. First, NCES 
must place greater emphasis on transforming raw data into information useful 
to policymakers, educators, researchers, and the general public than it does 
today. Accomplishing this goal will require that the relationships between 
NCES and data providers and between NCES and data users change signifi- 
cantly. During the next 5 to 10 years, the distinctions among these three par- 
ties— NCES, data providers, and data users — will become increasingly blurred, 
and their communications will probably become much more interactive, con- 
tinuous, and two-way, with all three parties actively and simultaneously 
engaged in survey design, data collection, analysis, interpretation, presenta- 
tion, and dissemination. Although technology will help pave the way for this 
transition, considerable conceptual thinking will also be required to take full 
advantage of the technological opporbmities. 

Second, in order to be more responsive to the 
demands for information about education, NCES 
will need to broaden its conception of what consti- 
tutes "data" and strategies for their collection. 

Traditionally, NCES has concentrated on designing 
and conducting surveys asking "closed-ended" 
questions that lend themselves to rapid, well- 
defined quantification. Although such surveys are 
likely to remain the hallmark of NCES's data collec- 
tion activities for some time, the agency will need to 
pay more attention to how to supplement these data 
with various forms of "prequantified" material and 
observations. Technological developments will per- 
mit inexpensive collection of increasing amounts of textual, visual, and auditory 
data as integrated supplements to surveys. This capacity should make it easier for 
researchers to ask questions and explore subjects that they did not foresee when 
designing the survey, thus enriching analytic power and reducing the expense of 
designing and conducting new surveys to examine unanticipated concerns. 

In addition to these two major themes, this effort led to five important conclu- 
sions about future directions for NCES. First, NCES should strive to produce 
information that addresses more immediate and specific policy concerns. While 
the agency's role in monitoring and describing major long-term trends in educa- 
tion must not be compromised, this role will assume even greater importance if 
the agency can also contribute in a timely way to more focused policy debates. 
The widespread emphasis on education reform during the past 10 years has 
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spawned a large number of different strategies for improving education. As a 
result, policymakers at all levels — national, state, and local — want to know 
more about what has and has not been accomplished. 

Second, surveys yielding better information that 
bears directly on the practices of teaching and 
learning would significantly enhance the contribu- 
tion of NCES to both policy debate and research. 
Since current surveys produce scant data on the 
specific content of curriculum, the nature and fre- 
quency of discrete classroom activities, the practices 
of teachers, or the kinds of tasks students perform in 
order to learn, the classroom remains largely a 
"black box" that defies clear understanding and 
precise strategies for improvement. Without a clearer 
understanding of what constitutes effective class- 
room practices, it will be difficult to do more than 
simply describe what kinds of education reforms 
have been implemented. Whether they have, in fact, 
improved teaching and increased learning will 
remain unknown. 

Third, survey designers should consider more 
carefully strategies that will permit integrated 
analysis of the interrelationships among education inputs, processes, and out- 
comes. Although existing surveys do an excellent job of providing nationally rep- 
resentative descriptive data on many im.portant aspects of education, they do not, 
however, lend themselves very well to reliable causal analyses that might 
increase knowledge about what works and why. The descriptive power of nation- 
al data must be preserved, but there are promising new designs emerging, which, 
if selectively incorporated into national surveys, might generate more robust con- 
clusions about the relative effectiveness of various educational practices. 

Fourth, NCES should make better use of data already collected and maintained 
by others. Doing so will help NCES simultaneously accomplish three aims; 1) 
expand the amount and type of data it collects; 2) adopt a wider range of data col- 
lection and analytic methods; and 3) function within the tight resource constraints 
that are certain to affect almost all federal agencies. Previously, NCES has pur- 
sued such a strategy with some success — for example, through the Common Core 
of Data (CCD) for elementary and secondary education and the Integrated 
Postsecondary Education Data System (IPEDS); however, high standards for data 
quality, especially comparability and reliability, have frequently forced the 
agency to collect new data that were already available in a somewhat different 

BEST COPY AVAILABLE 
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form or from a different time period. Without doubt, NCES must maintain its 
data quality standards, but increasing cooperation with states and localities, com- 
bined with rapidly improving data management technology and communica- 
tions, should create opportunities for the Center to do a better job of streamlining 
and coordinating data collection. 

Fifth, NCES will need to place increasingly greater emphasis on dissemination. 
Data and information are only as valuable as the breadth, quality, and timeliness 
of the uses made of them. Electronic storage media (data tapes and compact 
disks) and printed publications will surely remain the cornerstone of the agency's 
strategy for distributing data, tabulations, and the results of analysis. However, 
NCES should pay more attention to clearinghouse and brokerage functions, as 
well as effective use of electronic networks. 

These themes and conclusions do not represent radical departures from the major 
path NCES has been pursuing in recent years. Indeed, as the following section 
illustrates, they are well suited to building on the foundation of core functions, 
operating principles, and programs of work that support the current agency. 
Nevertheless, serious attention to these ideas will almost certainly produce 
important differences in what the agency now does and how it does it. 



Quilding the future on the current foundation 

In 1986, the Panel to Evaluate the National Center for Education Statistics, a 
group created under the auspices of the National Academy of Sciences (NAS), 
reported on the results of its 2-year assessment of the mission and effectiveness of 
NCES (Levine, ed. 1986). The Panel was created to address the widespread per- 
ception that the existing agency had not yet developed "the image and the reali- 
ty of a competent and objective major statistical organization serving the wide 
need for statistics about education in the United States" (Levine, ed. 1986, p. 13). 
To address such problems as quality of data, timeliness, conceptual obsolescence, 
and insufficient funding and staff, the Panel made many important recommen- 
dations, including the following: 

■ Clearly establish and define the Center's role in ensuring the availability 
of data needed to describe the condition of education in the United States; 

■ Improve the compilation of education program, staff, and financial data 
from the states, including developing closer collaboration with the states 
to ensure that the Center's program of work meets both NCES and state 
requirements for usefulness, relevance, quality, and reliability; 



From Data to Information 



^ 2 - 




■ Strengthen the Center's methodological and technical capacity through 
more systematic use of outside expertise in the Advisory Council of 
Education Statistics, as well as ad hoc advisory groups; 

■ Develop, publish, disseminate, and implement standards to guide all 
phases of the Center's work, including establishing an office of statistical 
standards headed by a chief statistician; 

■ In collaboration with the states, assess and improve the quality, consis- 
tency, and reliability of data obtained from state and local agencies, from 
institutions of higher education, and from other sources; and 



■ Institute a publications policy that clearly distinguishes between different 
types of reports — for example, statistical summaries and digests, analytic 
reports, descriptive reports, and reports on methodology — and develop a 
schedule of fixed release dates for selected key education statistics. 



Jhereas 10 years ago, the 
future of NCES depended 
on rectifying fundamental 
weaknesses, today the 
agency's future can build 
on a strong foundation. 



Ten years later, with the direction provided by the 
Panel, strong leadership at NCES, and support from 
Congress and the larger education community, 
NCES is much stronger and has become a widely 
respected statistical agency. The agency has signif- 
icantly strengthened its core functions; operates 
under well-defined guiding principles and high 
standards for data collection, analysis, and report- 
ing; and has established a clear program of work 



for reporting on the major aspects of education in the United States and other 



nations. Whereas 10 years ago, the future of NCES depended on rectifying fun- 
damental weaknesses, today the agency's future can build on a strong founda- 



tion. 



The Core Functions of NCES 

The National Academy of Sciences' Committee on National Statistics defines the 
principal purpose of a federal statistical agency as "the compilation and analysis 
of data and the dissemination of information for statistical purposes" (Martin and 
Straf, eds. 1992). NCES adheres to this primary purpose by organizing its work 
around three core functions: 

1) Survey Design and Data Collection 

2) Information Production — data analysis, translation, and interpretation 

3) Dissemination 
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Since national surveys are the primary means for 
NCES to collect data on education, during the 
past 10 years, the Center has devoted much 
effort to improving survey design and data col- 
lection. For instance, following the recommenda- 
tions of the 1986 NAS Evaluation Panel, it has 
developed and implemented various strategies 
to improve data quality, to detect and reduce 
error, and to expedite data collection. The Center 
has also significantly improved the sophistica- 
tion and efficiency of its sampling methods, 
increased its use of computer-assisted telephone interviewing, and systematical- 
ly assessed the quality of data generated in its national surveys. 

Moreover, NCES has strengthened and substantially expanded its capacity to 
analyze data. Rigorous statistical standards now govern all aspects of its analytic 
function, from simple tabulations to the most sophisticated multivariate analy- 
ses.^ The Center routinely applies procedures for quality control to all of its sur- 
veys, which include analyzing data quality, eliminating unacceptable error, and 
producing methodological and descriptive summary reports before releasing sur- 
vey data for public use. 

Finally, NCES has greatly expanded and improved its dissemination function. 
Toward this end, the agency has developed and implemented publication stan- 
dards that now guide the production of NCES reports and the release of public 
use data files. A central new feature of the agency's dissemination function has 
been developing strict policies for protecting the privacy of participants in NCES 
surveys. The Center not only applies safeguards to the data released to the public 
but also requires that its analytic contractors follow strict requirements for limiting 
access to prerelease data files and for maintaining the confidentiality of survey 
respondents. Failure to adhere to these requirements carries stiff fines, as well as 
the possibility of imprisonment. 



CES organizes its work 
around three core 
functions: 

1. Survey Design and 
Data Collection 

2. Information Production 

3. Dissemination 
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Operating Principles 

In carrying out these core functions, NCES adheres to three operating principles:"^ 



1) Produce information that is policy relevant, while maintaining strict 
impartiality, institutional independence, and neutrality with respect to 
programmatic effectiveness; 

2) Maintain credibility with users of its data, analysis, and publications; and 



3 ) 



Maintain trust among those who provide data, including individuals, 
institutions, and public and private agencies. 
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NCES's program of work must be guided by the issues and requirements of 
public policy and federal programs, while scrupulously avoiding specific policy 
recommendations or identification with particular policy agendas or ideologi- 
cal perspectives. This principle is perhaps easiest to achieve when the Center 
performs its responsibilities for providing data to others for analysis or when it 

produces tabulations and descriptive reports. 

When NCES engages in analysis or interpreta- 
tion, however, it must exercise greater care to 
remain policy neutral while still contributing 
relevant information to policy debates. 

Attention to this principle has important impli- 
cations for charting future directions for NCES. 

The suggestions that the Center address more 
immediate, specific policy concerns and that it 
develop survey designs more strongly suited 
for evaluation of what works could lead it 
beyond the boundaries of policy relevance into 
policy statements and evaluation. This, in turn, could jeopardize its position of 
impartiality. Deriving greater policy benefit from data and information produced 
by NCES, therefore, must proceed with great care. 

Attention to this first principle also contributes to realizing the second, credibility 
with users of NCES data and information. However, credibility depends on more 
than policy relevance and impartiality. It also derives from confidence in the rigor 
of survey design, the quality of the data, the strength of analysis, and the accessi- 
bility and usability of its products, publications, and services. Here again, as NCES 
considers making greater use of data collected and maintained by others, it will 
need to guard against undermining its credibility with users who now depend on 
the Center's increasing emphasis on methodological rigor and data quality. 

Finally, the success of NCES as an information agency rests on the trust it 
engenders among those who supply it with data. Protecting the privacy of sur- 
vey participants is a key aspect of maintaining this trust, and integrating new 
types of data into national surveys will pose challenges for assurances of confi- 
dentiality. Use of video and audio data — for example, taping teachers in the 
classroom — will require close scrutiny of this issue. Confidentiality, however, is 
not the only condition for securing trust among data providers. Suppliers of 
data also need to be confident that the information being requested is truly 
needed, that it will be tabulated and analyzed accurately, and that providers 
will be given opportunities to correct errors or clarify ambiguities. Pressures for 
greater timeliness or more direct electronic access to decentralized, raw data 
files may undermine the confidence of data providers in the absence of explicit 
attention to new strategies and safeguards. 
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Program of Work 

NCES has organized its current program of work around seven major topics:^ 

■ Elementary and Secondary Education 

■ Postsecondary Education 

■ Educational Assessment 

■ National Longitudinal Studies 

■ International Comparative Studies 

■ Vocational Education 

■ Libraries 

Information on each of these topics is produced from a variety of surveys and 
studies, several of which supply data to more than one topical area. Some of the 
surveys, such as the CCD (on elementary and secondary schools and school dis- 
tricts), are designed as a census of the universe of respondents, which then serves 
as a sampling frame for more in-depth cross-sectional or longitudinal surveys on 
smaller samples of the population. The Schools and Staffing Survey (SASS), for 
example, collects detailed information on teachers and administrators in a sam- 
ple of schools drawn from the CCD.^ In other instances, a large comprehensive 
survey provides the basis for a more intensive study of a subset of respondents. 
In this vein, the National Postsecondary Student Aid Study (NPSAS) — a nation- 
wide survey of students enrolled in postsecondary institutions — provides the 
basis for more targeted longitudinal studies of students who are starting postsec- 
ondary education, the Beginning Postsecondary Students (BPS) Longitudinal 
Study, and of students who have completed a baccalaureate degree or higher, the 
Baccalaureate and Beyond (B&B) Longitudinal Study. 

Central to the NCES program of work are various surveys and studies 
designed to assess the knowledge, skills, and performance of American students. 
For instance, the National Assessment of Educational Progress (NAEP), which 
conducts assessments of reading, mathematics, writing, science, history, and 
geography for samples of students enrolled in elementary and secondary educa- 
tion,7 is probably the best known of these efforts. In addition to NAEP, NCES also 
provides other data on student performance through transcript studies (at both 
the secondary and postsecondary levels); through the National Adult Literacy 
Survey, which examines adults' ability to use prose, documents, and mathemat- 
ics in a variety of commonplace daily activities; and through international assess- 
ments that provide comparative information about student performance in the 
United States relative to that of other countries. 
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Finally, the Center conducts several long-term longitudinal studies designed to 
track students' paths through school and into subsequent stages of working and 
family life. These have included such studies as the 1980 High School and Beyond 
(HS&B Study), the National Education Longitudinal Study of 1988 (NELS;88), 
and the Early Childhood Longitudinal Study (ECLS), which is still in the plan- 
ning and testing stage and is expected to begin with a kindergarten class in school 
year 1998-99. 

These surveys now contribute to approximately 100 publications that NCES pro- 
duces each year, including descriptive reports, analysis reports, methodological 
reports, issue briefs, and a variety of other documents. Three of these docu- 
ments— the Digest of Education Statistics, Projections of Education Statistics, and Tlw 
Condition of Education — annually provide a broad national overview of education 
at all levels in the United States. 

In summary, during the past 10 years, NCES has been engaged in a process of 
steady development and improvement. In 1996, NCES is a viable and credible sta- 
tistical agency, applying high standards to the provision of information on the 
condition of education in the United States and the nation's progress toward 
improving mastery of knowledge and skills among all its citizens. With this 
strong foundation, the agency is now well positioned to pursue some new direc- 
tions that will enhance its ability to produce important information for policy, 
research, and practice in American education. 




EW DIRECTIONS IN INFORMATION FOR POLICY, 
RESEARCH, AND PRACTICE 



In considering how NCES can best chart a course over the next decade that will 
capitalize on the foundation of work already in place, it is useful to consider its 
contribution to three domains of education: policy, research, and practice. These 
domains are by no means mutually exclusive; in fact, they overlap and interact in 
important ways. There are, however, information needs that are either unique or 
more dominant in each, and it is therefore instructive to consider the following 
questions individually: 
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How can NCES best contribute information to discussions of education 
policy at the national, state, and local levels? 

How can NCES contribute information that will support significant 
research on education effectiveness and improvement? 
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■ How can NCES contribute information that supports practice — i.e., the 
"front-line" activities that develop knowledge and skill in the nation's 
students? 

A fourth question constantly weaves through these first three: what advances in 
methodology and technology can assist NCES in providing useful information to 
each of these domains? This section addresses each of these four questions in turn. 



Information for Policy 

The agenda of NCES is, in the first instance, greatly influenced by public policy 
issues and the requirements of federal, state, and local programs affecting education. 
Information contributing to policy debates can assume at least three major forms: 

1) System indicators that describe the functioning of the education enterprise, 
broadly and over the long term; 

2) Implementation indicators that describe the breadth and depth of the execu- 
tion of policies and practices; and 

3) Effectiveness indicators that describe the results achieved by students and 
educational institutions and programs. 

Although system indicators have been a long-standing focus of NCES, there are 
potentially important new developments for the agency to consider. NCES sur- 
veys have included information on the implementation of some generic policies 
and practices, but specific federal and state policy initiatives have not been exam- 
ined. Surveys have also included measures of student outcomes — the NAEP is 
the best known example; however, these measures typically carmot be directly 
linked to particular policies or educational practices to permit rigorous assess- 
ments of effectiveness. What are some possible new directions for NCES to con- 
sider with respect to each of these three types of indicators? 

System Indicators 

Data that portray the major aspects of the American education enterprise, both 
cross-sectionally and over time, form the core of the mission and functions of 
NCES. Reporting basic descriptive information on students, faculty and other 
staff, institutions and governing districts, and education finances must continue 
to be the primary focus of NCES and should not be compromised by new initia- 
tives. The authors contributing to this examination of new directions for NCES 
are unanimous on this point: the primary purpose of NCES is and should remain 
representatively describing and documenting the condition of education in 
America and other nations. 
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Percentage of High School Graduates Who Earned the 
Recommended Units in Core Courses, by Race/Ethnicity: 
1982 , 1987 , 1990 , and 1992 




SOURCE: U.S. Department of Education, National Center for Education 
Statistics, The Condition of Education 1995, Washington, D.C., 78-79. 



This basic description of the educa- 
tion system can, of course, be 
improved, and several of the 
papers included in From Data to 
Information: New Directions for the 
National Center for Education 
Statistics, offered suggestions.® 
Among the kinds of system infor- 
mation the authors would like to 
see developed are the following: 

■ Detail on curriculum con- 
tent, including rigor and 
substance; 

■ Detail on the nature and fre- 
quency of particular teaching 
practices, especially those 
which research indicates are 
effective; 



■ Attention to the nature and frequency of student behavior that reflects 
engagement in learning; 

■ Resource allocation at the institutional and classroom level; 

■ Measures of teacher quality and the ways in which teachers apply their 
knowledge and skills in the classroom; 

■ More contextual information on postsecondary institutions, especially 
their objectives in awarding student financial aid and improved coverage 
of proprietary institutions; 

■ More attention to the interaction between education and work; and 

■ More attention to governance issues, particularly new organizational and 
oversight arrangements. 

In many respects, these recommendations represent requests for "finer grain" in 
the descriptive data presently collected by NCES. In some instances, this aim can 
be accomplished by asking for more detailed information; in other instances, col- 
lecting and reporting existing data at lower levels of aggregation (the classroom, 
for example, rather than the school or school district) will be necessary. 
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Implementation Indicators 
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The widespread attention on education reform during the past 10 to 12 years 
has spawned a number of new policy initiatives at the national, state, and local 
levels. Congress periodically revises such mainstay education legislation as the 
Elementary and Secondary Education Act, the Higher Education Act, or the 
Individuals with Disabilities Education Act. Additionally, it undertakes new 
education policy initiatives such as GOALS 2000 or the School-to-Work 
Opportunities Act. States have also initiated many new policies to strengthen 
elementary, secondary, and postsecondary education. These have included 
changes in the requirements for high school graduation, new teacher certifica- 
tion regulations, modifications to postsecondary admissions standards, and 
new policies on college tuition and student financial aid. 

Traditionally, NCES has not monitored the implementation of specific federal or 
state legislation. At the national level. Congress has typically provided for inde- 
pendent assessments or evaluations of education legislation, such as the National 
Assessments of Vocational Education, which have been conducted approximate- 
ly every 5 years. While these national assessments make extensive use of NCES 
data, they also conduct independent surveys that focus more particularly on key 
features of the legislation being examined. 

Several of the authors involved in this project have urged NCES to monitor some 
of the key policies and practices that have emerged from federal, state, and local 
legislation during the 1980s and 1990s. ^ It should be emphasized that they are not 
recommending that NCES assume responsibility for evaluating particular legis- 
lation, because they believe this function should continue to rest elsewhere. 
Rather, they are urging NCES to examine policies and practices that became more 
generic as they have been adopted and implemented through various federal, 
state, and local initiatives and are, therefore, no longer associated with any single 
piece of legislation. Some specific examples include the following: 

B Curriculum content standards and measures of student or institutional 
performance; 

B Length of the school day or year; 

B Requirements that students complete particular courses (for example, in 
math, science, or foreign language) or accumulate a minimum number of 
credits for graduation; 

B Participation in a variety of "work-based" learning opportunities, includ- 
ing apprenticeship, cooperative education, tech-prep programs, or school- 
based enterprise; 

B Operation of charter schools; 
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■ Prevalence and nature of home schooling; 

■ Availability and use of school choice; 

■ Participation in reform networks, such as the Coalition of Essential 
Schools, Accelerated Schools, or High Schools That Work; 

■ Changes in affirmative action policies; 

■ Changes in postsecondary admission requirements; 

■ Prevalence of state takeovers of local school districts or other forms of 
state inferv'ention in financially troubled localities; and 

■ Changes in state policies affecting postsecondary tuition or student 
financial aid. 



More attention to such issues by NCES would help ensure that its data are policy 
relevant, while still leaving responsibility for policy evaluation to independent 
studies and other agencies in the Department of Education. 

Effectiveness Indicators 



jn the current environ- 
ment, policy analysts 
often face a frustrating 
choice: asking the right 
question with weak 
methodology and data 
that were not collected 
specifically for that pur- 
pose, or asking a much 
less important question 
with sound methodology 
and specially tailored 
information. 



In addition to information on how to implement 
policies and practices, policymakers would also 
like better information on their effectiveness. Even 
though it is useful to know how widespread the 
adoption of a particular strategy for improving 
education has been, it is even more useful to know 
how well it has worked, and why or why not. 
This, of course, is a primary aim of most policy 
evaluation, as well as many research projects. 

Much of the credibility of NCES rests on its clear 
separation from policy evaluation and research 
on education impacts and outcomes. Although 
NCES contributes essential data and information 
to these efforts, it remains well removed from the 



conduct of any of these activities. This separation of functions contributes to the 
neutrality and objectivity that NCES must maintain as the nation's primary sta- 
tistical agency for education. The impartial character of NCES must be preserved. 
Consequently, any initiative to make NCES surveys more conducive to assess- 
ments of policy effectiveness must proceed with great care. 



Why consider such a course at all? First, there is potentially a substantial payoff 
from better integrating the nationally representative features of NCES surveys 
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with the more rigorous but also more narrowly 
circumscribed designs of policy impact studies. 
In the current environment, policy analysts 
often face a frustrating choice: asking the right 
question with weak methodology and data that 
were not collected specifically for that purpose, 
or asking a much less important question with 
sound methodology and specially tailored infor- 
mation.^^ Clearly, answering important ques- 
tions with sound methods and precise informa- 
tion is more likely to improve education and the 
policies that support it. Combining the repre- 
sentative power of national surveys with the 
methodological rigor of experimental design 
would help realize this objective. 
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je primary benefit of 
integrating methodoiogi- 
caiiy rigorous effective- 
ness assessments with 
nationaiiy representative 
surveys iies in increasing 
tite usefuiness of these 
two activities beyond tite 
resuits obtained when 
they are conducted 
independentiy. 




Second, there may be significant cost savings from integrating some impact eval- 
uations with national surveys. Both kinds of efforts are quite costly. It is not 
unusual for a national survey to cost in excess of $10 million, and the more rigor- 
ous policy evaluations adopting experimental design frequently cost as much or 
more. Both efforts often collect similar kinds of data at approximately the same 
points in time, sometimes even from the same respondents. Eliminating this 
duplication would not only reduce costs but also alleviate some of the burden on 
respondents participating in national surveys and evaluations. 



Cost savings aside, the primary benefit of integrating methodologically rigor- 
ous effectiveness assessments with nationally representative surveys lies in 
increasing the usefulness of these two activities beyond the results obtained 
when they are conducted independently. National survey data would more 
directly and authoritatively address questions about policy effects; impact 
studies would be conducted in a nationally representative context that would 
increase the likelihood that study results could be generalized. 

To achieve this result, NCES should carefully consider piloting the inclusion of 
an experimental study in one of its national surveys. Any of the longitudinal 
surveys now under way are potential candidates, including SASS, ECLS, or the 
longitudinal spin-offs of NPSAS. 



What should be the focus of experimental studies imbedded or linked to 
national surveys? Clearly, the choice must be considered carefully, with ample 
input from interested policymakers, researchers, and educators. Given the mis- 
sion of NCES, focusing on a particular type of educational practice would 
probably be more appropriate than on an assessment of a specific legislative 
program. One possibility, for example, would be to conduct a careful study of 
the consequences of homogeneous versus heterogeneous grouping of students 
by academic ability. 
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Information for Research 



Researchers are heavy users of information produced by NCES, and while the 
boundaries between policy and research are fuzzy, the interests of the research 
community deserve some separate attention. In the papers produced for this pro- 
ject, three themes emerged as priorities for focusing NCES's contribution to 
research over the next decade: 

1) Teaching and Learning — illuminating more clearly what actually happens 
in the classroom; 

2) Education Production — clarifying the processes of transforming educa- 
tion resources into student, program, and institutional outcomes; and 

3) Education Outside the Classroom— depicting what and how learning 
occurs beyond the walls of the traditional classroom in homes, work- 
places, and the community at large. 

Though not exhaustive, this list provides some important directions for NCES to 
consider. In the next section, each topic will be briefly discussed. 

Teaching and Learning 

Much of the business of education occurs in the nation's classrooms — elementary, 
secondary, and postsecondary — yet national surveys presently tell us relatively 
little about what actually takes place at the classroom level. Currently, good infor- 
mation is available about the different types of courses taught (at both the sec- 
ondary and postsecondary levels), but there is little or no nationally representa- 
tive detail on the content of the curriculum or how it varies among classrooms, 
institutions, or states. Similarly, not much data are available on teaching prac- 
tices, either the range of strategies adopted by faculty or the frequency of their 
use. Finally, most surveys do not offer much description of what students do to 
facilitate or impede learning in the classroom.^^ 

While richer information on these three aspects — curriculum, pedagogy, and stu- 
dent behavior — would be useful in and of itself, the greatest benefit to research is 
likely to be achieved when information on all three is simultaneously available at 
the individual classroom level. That is, ideally researchers would want to exam- 
ine how these three aspects of classroom activity interact and to understand how 
they relate to various types of education outcomes. In this way, more can be 
learned about what works and why in the daily business of education. 

To realize this objective, one implication for future NCES surveys is clear; survey 
designs need to pay more attention to using the classroom as a unit of analysis. 
Additionally, the designs should strive to produce an integrated package of infor- 
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mation on curriculum content, teaching prac- 
tices, student behaviors, and student learning 
outcomes. It is not sufficient, for example, to 
simply expand transcript studies to include 
more information on course content; rather, 
expanded information on course content must 
be linked to other data on teaching practices, 
student behavior, and student achievement. 

In addition to the question of what kinds of 
information on classroom activity can best 
advance future research, there is also the issue 
of how best to collect it. Traditionally, to obtain 
data on classroom activities, NCES has asked 
respondents questions through paper question- 
naires or telephone interviews. Thus, to the 
extent that current surveys yield information on 
teaching practices or student behavior, they rely mainly on self-reports. 



0ne implication for future \ 
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An alternative to collecting information through respondent self-report is direct 
observation by trained researchers. Until recently, direct observation has been a 
very expensive alternative, indeed prohibitively so for large-scale surveys involv- 
ing thousands of respondents. However, recent technological and methodologi- 
cal advances are making direct observation, as well as the collection of source 
materials, more feasible.!^ Video is one of the most promising strategies for link- 
ing direct observation to more traditional survey techniques, and NCES is using 
this technique for the first time in designing the Third International Math and 
Science Survey (TIMSS).l^ 

Video, of course, is not an especially new tech- 
nology. What is new, however, is its rapidly 
growing capacity to store large amounts of video 
information inexpensively in digital form that 
enables fast retrieval and analysis. Additionally, 
researchers are making steady progress in devel- 
oping analytic techniques that simplify and 
accelerate transforming video information into 
coded data suited for analysis using quantitative 
methods. Video, therefore, can add significantly ; 
to the richness and analytic potential of a survey, 
since it reduces the need to anticipate all of the 
questions the survey must ask of respondents. 

As researchers observe video records, they can formulate completely new vari- 
ables that may not have been considered in the design phase of the survey. In the 
past, such new formulations usually required asking respondents follow-up 
questions or designing a new survey, if such avenues were pursued at all. 



Wideo is one of the most 
promising strategies for 
linking direct observation 
to more traditional survey 
techniques, and NCES is 
using this technique for 
the first time in designing 
the Third International 
Math and Science Survey 
(T1MSS). 
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Closelv related to this kind of use of video tech- 
nology is the increasing capacity to collect, store, 
and analyze large amounts of textual information. 
For example, if researchers need better informa- 
tion on the content of textbooks or other printed 
materials used in classrooms, it is now possible to 
optically scan samples of these classroom docu- 
ments for subsequent coding and analysis. As 
with video images, electronically storing and 
retrieving large amounts of textual information is 
relatively inexpensive. 

These advances in storing and analyzing large amounts of what is essentially 
"prequantified" data promise to integrate survey research with case study 
methods, and represent research strategies that until now have been pursued 
independently of one another, each with its own strengths and weaknesses. 
This integration has the potential to link the representative statistical power of 
survey design with the richness and variety of case study information, simulta- 
neously obtaining the best of both worlds. 

Expanding surveys to include systematic collection of prequantified visual, tex- 
tual, and even auditory information could produce significant new contributions 
to research on teaching and learning. Consequently, NCES should carefully con- 
sider how best to capitalize on its initial experience with this strategy in TIMSS, 
with special attention to adapting the approach to other surveys such as SASS, 
ECLS, or the longitudinal spin-offs of NPSAS. Additionally, the use of video in 
national surveys might prove especially beneficial if it were initially combined 
with efforts to imbed experiments in national surveys. The combination of these 
two methods targeted on analyzing the effectiveness of particular teaching inter- 
ventions, for example, could yield very useful and robust results. 

Education Production 

Better understanding the interactions among curriculum, pedagogy, and student 
behavior in the classroom is an important piece of a larger set of research ques- 
tions— how dollars are allocated (to localities, institutions, and classrooms), trans- 
formed into various resources, organized into programmatic and teaching strate- 
gies, and used to produce increases in students' knowledge and skills. In short, 
NCES data could play a much more significant role in expanding knowledge 
about how to better use education resources to improve student performance, 
thereby improving the overall process of education production. 

Achieving this goal will require some changes in the way NCES currently collects 
data on the financing of elementary, secondary, and postsecondary education. At 
present, there are two main surveys collecting financial data, CCD at the elemen- 
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tary and secondary level and IPEDS at the post- 
secondary level.^^ With respect to financial infor- 
mation, both of these surveys focus primarily on 
providing detail on revenues and expenditures, 
for local school districts in the case of CCD and 
for individual institutions in the case of IPEDS. 
Both surveys are designed to collect financial 
data primarily from an accounting perspective 
and are not now well suited for cost-benefit 
analysis of educational programs or cost-effec- 
tiveness analysis of particular teaching strategies. 
Neither provides information on the allocation of 
resources at the classroom level. 
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Providing data that better inform understanding of the production process of 
education would be aided by NCES expanding its present focus on finance to 
embrace a broader concentration on the economics of education. This larger con- 
ception would aim to integrate data on finance with other data on education 
processes and practices, as well as student outcomes. Additionally, new kinds of 
economic data would be required. Rather than needing more detail on expendi- 
tures for such general functions as administration, instruction, maintenance, or 
capital outlay, researchers would want to obtain data on the costs of specific 
types of staff, different kinds of school improvement strategies, alternative 
teaching strategies, and so on. They would also want to know more about the 
costs of different kinds of course content, equipment, instructional products, and 
assessment. In short, rendering the process of education production more intel- 
ligible depends on moving 

beyond traditional concerns Public Education Revenue Per Student, 1930~93 

about the distribution and 
expenditure of dollars to a more 
careful examination of how to 
transform dollars into effective 
teaching and learning in the 
classroom. 



Three strategies for improving 
NCES data on education finan- 
ces would help accomplish this 
goal. First, what constitutes use- 
ful financial data needs to be 
reconsidered, with special atten- 
tion to better information on 
unit costs and transforming dol- 
lars into education processes 
and practices. Second, data will 
be needed at the classroom 




SOURCE: U.S. Department of Education, National Center for Education 
Statistics, The Condition of Education 1995, Washington, D.C., 148-49. 
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level; information on districts or institutions is not likely to contribute much to 
this kind of research. Third, it must be possible to link these financial data to 
other data on teacher characteristics, classroom practices, student demograph- 
ics and behavior, and learning outcomes. Without this kind of integrated data 
about how education occurs, understanding more precisely how to efficiently 
allocate resources for education will continue to elude researchers and policy- 
makers. 



Education Outside the Classroom 

Although elementary, secondary, and postsecondary classrooms are the centers 
of formal education in America, it is widely understood that much learning also 
takes place outside the classroom in the home, the workplace, and the communi- 
ty at large. However, we know relativelv little about what or how learning occurs 
in these settings, nor do we know much about how learning in these places inter- 
acts with learning in the classroom. Moreover, given that most Americans spend 

only 12 to 16 years in formal schooling but anoth- 
er 50 years or so learning in these informal envi- 
rons, a thorough description of the condition of 
education in America would require closer atten- 
tion to the learning that transpires beyond class- 
room walls. 

NCES surveys have already paid some attention to 
nonschool settings. At the present time, probably 
the largest of such efforts is the Early Childhood 
Longitudinal Study (ECLS), which will begin by 
focusing on the preschool lives of a cohort of chil- 
dren who will be followed over their early years of 
development. Additionally, other longitudinal 
studies, such as HS&B and B&B, have collected 
data on respondents' experiences in the work- 
place. Information on labor market participation, 
however, has been limited primarily to data on 
types of labor market outcomes - for example, earnings, duration of employment, 
and types of occupation — rather than systematically examining how learning 
occurs in the workplace or the degree of congruence between learning goals in 
schools and education requirements on the job. 
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Comprehensively surveying learning that occurs outside the classroom is a tall 
order, and NCES should approach this task incrementally. One place to focus an 
expanded examination of informal learning is on the workplace and the strategies 
adults use to maintain and upgrade the knowledge and skills needed to remain 
productive, activelv engaged workers. Such a focus is more important than 
ever, given the changes that are occurring in today's work world. These changes 
include not only rapidlv developing new technologies but also major shifts in the 
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attachments and relationships between employers 
and employees. As the likelihood of lasting 
employment with a single employer becomes 
increasingly tenuous in the modern economy, 
individuals must assume ever greater responsibil- 
ity for nurturing their own careers and continuing 
employability. How working adults will meet this 
responsibility in the future poses important new 
challenges for the nation's systems of education. 

Increased attention to learning through and for 
work could begin with the following steps. First, it 
is important to learn more about the knowledge 
and skills needed for long-term success in the labor market. Are these require- 
ments consistent with the academic and vocational goals of formal education, and 
how well does the formal education system produce the desired prerequisites? 
Second, NCES could pay closer attention to how learning occurs in the work- 
place; whether the process differs in important ways from learning in the class- 
room, and whether the two complement or reinforce one another. Third, NCES 
could enrich work-related data in its current longitudinal surveys, concentrating 
especially on better information about what people do on the job, what con- 
tributes to their successes and failures, and how they use or do not use school- 
based learning to perform and advance. 

As part of its own mission, NCES could independently address all three of these 
issues. Alternatively, the agency may want to explore opportunities for collabo- 
rating with the Department of Labor and its surveys of employers and employ- 
ees. Data collected by the Labor Department tend to provide greater detail on 
labor market participation, while being relatively deficient on education vari- 
ables. Better coordination or integration of the two Departments' survey efforts 
could yield some important benefits. 
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Information for Practice 

Most teachers and administrators are accustomed to viewing data as something 
to be reported to others. For example, they report daily attendance to central 
offices to document federal, state, and local funding systems. They submit grades 
for report cards to students and parents and for recording on student transcripts, 
which in turn are reported to postsecondary admissions offices. They administer 
standardized achievement tests for state assessments and college admissions. 
Rarely, however, do teachers and administrators use data directly themselves to 
improve their own programs and practices. One consequence of this outlook on 
data is that most practitioners do not make much use of the information pro- 
vided by NCES. An important challenge for NCES, therefore, is significantly 
increasing the value and utility of its data for local teachers and administrators. 
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At least three strategies for providing better infor- 
mation for practice offer some important opportuni- 
ties for NCES: 

1) Benchmarking— helping local practitioners make 
comparisons against established norms; 

2) Networking— linking practitioners with other 
practitioners and helping them discover more 
quickly who is doing what and where; 

3) Practitioner-Based Research and Self-Reflection — 
engaging practitioners in systematic inquiry 
through NCES surveys and related research. 



Benchmarking 

"How well am I performing?" This is an appropriate question for any professional 
concerned with improving practice and increasing students' mastery of knowledge 
and skills. For most educators, however, it is a difficult question to answer in any 
way other than in an impressionistic or anecdotal fashion. Until recently, education 
has not had enough success with helping schools, programs, and faculty to monitor 
their accomplishments or to use the results to improve what they do. 



Fortunately, as more and more states and localities develop new strategies for 
tracking performance and promoting school improvement, this situation is 
changing. "School report cards" are now produced annually in many states. 

Other states have developed systems of performance measures and standards, 
along with procedures for school improvement plans in districts that perform 
below state norms. Moreover, "keeping score" and using the results to assess the 
relative effectiveness of different kinds of school improvement strategies are core 
operating principles of several large consortiums, such as High Schools That Work 
under the auspices of the Southern Regional Education Board. 

NCES could make an important contribution to the continued development of 
these practices by improving the utility of its survey results as benchmarks for 
states and localities interested in knowing how their performance measures up in 
relation to others. A local school or school district, for example, could find out 
how well its record on student attendance or high school completion compares 
with a national or state norm. It could then further refine the comparison by 
examining such measures in a subset of districts or schools that are similar with 
respect to size or student demographics. In addition to making comparisons at a 
particular point in time, a local school or school district might also monitor its rel- 
ative performance over time. For example, is its success in reducing dropout rates 
proceeding at a faster or slower pace than in comparable districts or schools? 
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Median Total Time to Doctorate Degrees among U.S. 
Citizens and Permanent Residents, 1970-93 



There is nothing to prevent local- 
ities, or even individual teachers, 
from using current NCES data to 
establish these kinds of bench- 
marks. However, they need to 
work rather hard to do it. Finding 
the right data is not always easy, 
nor is determining whether the 
NCES estimate is comparable 
with a local statistic. Tailoring an 
NCES estimate to yield a compar- 
ison of "likes with likes" requires 
a knowledge of NCES data sets, 
as well as analysis techniques, 
that most practitioners do not 
have. Thus, there are significant 
barriers to transforming NCES 
data into useful benchmarks at 
the local level. 

However, there are at least three 
steps that NCES could take to 
make benchmarking easier for 
states and localities. First, in col- 
laboration with potential state and local user-practitioners, NCES could sys- 
tematically review its current dissemination activities with specific attention to 
how some aspects of the dissemination process could be modified to facilitate 
benchmarking. For example, there may be consensus on a relatively small set 
of indicators that NCES could publish annually in a succinct, accessible form 
with widespread local distribution. Such a publication might be similar to The 
Pocket Condition of Education, which NCES now produces annually, but it could 
be designed with local benchmarking specifically in mind. 




SOURCE: U.S. Department of Education, National Center for Education 
Statistics, The Condition of Education 1995, Washington, D.C., 46>47. 



Second, NCES could explicitly consider local benchmarking when designing select- 
ed surveys, including customized reporting of results to survey participants. At pre- 
sent, survey participants receive little or no direct benefit from taking part in NCES 
surveys, and the burden of doing so is often not trivial. Providing participants with 
a summary of where they stand on selected variables relative to others in the survey 
could be a useful serviceT^ Such a summary could take the form of a traditional 
printed report. Alternatively, NCES might want to explore new electronic strategies 
that could actually distribute some limited analytic capacity along with the data (see 
discussion below on technological innovations). 



Third, as NCES increases its capacity to provide information "on-line," it should 
consider strategies for developing and distributing analytic packages that enable 
state and local benchmarking. In other words, instead of simply making data avail- 
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able, NCES would also provide a menu of data 
analysis programs or routines that would enable 
practitioners to generate their own statistics 
quickly and easily, without requiring a sophisti- 
cated knowledge of the underlying methodology. 
Such a strategy would build on NCES's current 
practice of providing users with "table genera- 
tors," increasing both the kinds of analysis that 
users could perform and the ease of using the ana- 
lytic software. 

Networking 

Local teachers and administrators often want to 
know who else has experience with a particular 
school improvement strategy, type of curricu- 
lum, or teaching practice. Yet, systematically 
locating and communicating with other knowl- 
edgeable practitioners can be quite difficult; 
often it is not easy to find out who these individ- 
uals are or how to contact them. If NCES were to 
assume a greater role in monitoring the imple- 
mentation of more specific education policies and practices (see earlier section on 
Implementation Indicators), it could also facilitate networking among practition- 
ers. In addition to providing practitioners with information about the frequency 
with which a particular reform is being implemented and where it is being 
attempted, NCES could also match up interested parties and help them share 
information about their experiences. 

This kind of knowledge brokering would represent a new function for NCES, one 
that may not be completely in keeping with traditional perspectives on the appro- 
priate role of a statistical agency. Nevertheless, as NCES develops its presence on 
the Internet and the World Wide Web, this kind of service would be an obvious 
extension of its capacity to transform data into information valued by practition- 
ers. Moreover, when providers of data also have a direct use for similar informa- 
tion from others, they are much more likely to respond to NCES's requests in an 
accurate and timely fashion. Thus, NCES's ability to monitor implementation for 
policy purposes could well be enhanced by its also using the information to pro- 
vide an important service to teachers and administrators. 



iree steps that NCES could 
take to make benchmark- 
ing easier for states and 
localities: 

1. Review its current 
dissemination activities 
with specific attention to 
how some aspects of the 
dissemination process 
could be moditied to 
facilitate benchmarking. 

2. Explicitly consider 
local benchmarking when 
designing selected 
surveys. 

3. Consider strategies for 
developing and distributing 
analytic packages that 
enable state and local 
benchmarking. 




Practitioner-Based Research and Self-Reflection 

Until recently, surprisingly little has been known about the specific elements of 
high-quality teaching (and by extension, high-quality teachers). This lack of 
knowledge has contributed to much misinformation and misunderstanding 
about what it takes to be a good teacher, as well as confusion in the public policy 
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arena over the role of professional development in education reform. The status 
of national data on teachers reflects this state of affairs, with facts limited mainly 
to demographic characteristics and scant information available on the quality of 
practice or practitioners. 



A very promising development, therefore, is the recent effort on the part of the 
teaching profession to begin a systematic, sustained examination of what consti- 
tutes good teaching — specifically what teachers should know and be able to do to 
help students master high levels of proficiency. Exemplified by the work of the 
National Board for Professional Teaching Standards (NBPTS), as well as other 
organizations and state-level initiatives, this effort is forging some consensus 
about appropriate standards for defining advanced high-quality teaching. This 
work has led to the establishment of a voluntary system of certification for early 
childhood, elementary, middle, and secondary school teachers, including differ- 
entiation among a range of academic disciplines (for example, math, science, 
history and the social sciences, English, and vocational education). 



These developments create an important oppor- 
tunity for NCES not only to improve the data it 
gathers on the nation's teachers but also to con- 
tribute more directly to strengthening teaching.!^ 

This opportunity can be realized in two ways. 

First, as the work on teaching standards and cer- 
tification continues to evolve, NCES should be 
able to define a larger array of indicators of 
teaching quality to include in national surveys. 

At a minimum, these indicators should focus on 
measuring teachers' command of the knowledge 
bases and teaching methods that are being iden- 
tified as reflective of high-quality practice. 

Gathering such data could rely on traditional 
methods of written assessment or self-report. 

Alternatively, if NCES opts for further develop- 
ing video observation techniques, these methods could significantly enrich 
information on the condition of teaching nationwide.^O Furthermore, as more 
teachers choose to pursue national certification and as more certificates are 
awarded, national counts of teachers participating in and successfully complet- 
ing the process will assume greater value as indicators of teacher quality. 
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Second, as NCES pursues this first strategy for improving data on the quality of 
practice, it could actively engage practitioners in this process and create opportu- 
nities for more interactive research and development. For example, if written 
examinations (in the style of NAEP) or video observation become part of NCES's 
strategy for monitoring and reporting to the nation on teacher quality, this 
process could be designed to simultaneously benefit individual teachers par- 
ticipating in the surveys. This might be accomplished in several ways. The 
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results of written assessments could be returned to individual teachers. Groups 
of teachers could be assembled to review and constructively critique video seg- 
ments. Further, if the data gathering process included collecting mini-portfolios 
submitted by teachers, these could be systematically evaluated, with examples of 
best practices culled from the data and disseminated to teachers and teacher 
education institutions. In short, the business of collecting national data could 
begin to play a more direct role in professional development and the strategic 
improvement of teaching and schools. 



Further Considerations About Methodology and 
Technology 

Increasing the contribution of NCES to policy, research, and practice depends 
in part on closer attention to a number of methodological and technological 
opportunities. Some of these, such as imbedding experimental designs in 
national survevs or collecting prequantified data through video and optical 
scanning, have already been discussed. There are some additional considera- 
tions, however, that deserve special mention, including: 1) developments in 
using administrative records, 2) promising techniques for obtaining hard-to-get 
information or producing more finely tuned estimates, and 3) effective use of 
the Internet and World Wide Web. 

Administrative Records 

Much of the information sought by national surveys already exists, at least in 
an approximate form, in records maintained for administrative purposes by 
schools, postsecondary institutions, district offices, state agencies, and other 
public and private offices. Transcripts, for example, provide detailed informa- 
tion on courses attempted and completed, grades, credits earned, and scores 
from standardized tests. Personnel records contain data on teaching assign- 
ments, salaries, demographics, qualifications, and experience. And budget and 
accounting offices maintain extensive records on revenues and expenditures. 
To the extent that surveys can access and use these administrative records, 
they often can obtain information that is more accurate than the responses pro- 
vided by survey participants, often at significantly less cost. 

Several NCES surveys already rely heavily on administrative records for 
information. Some good examples are CCD, IPEDS, and the NPSAS. There are, 
however, two types of problems that have limited the usefulness of administra- 
tive records. First, the contents of the records may not meet acceptable standards 
of accuracy, consistencv, and comparability. Second, access to administrative 
records is often problematic, for a variety of reasons ranging from concerns about 
confidentiality to technical problems that may be as mundane as locating the 
right filing cabinet in the right office. 
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Technological advances in computing and electronic networking promise to 
reduce both of these problems considerably over the next decade, and NCES 
should be alert to opportunities to exploit new developments. First, electronic 
administrative records maintained in easy-to-use relational databases will 
increasingly become accepted practice among the nation's elementary, sec- 
ondary, and postsecondary education systems,^^ since they will be building 
administrative databases to satisfy their own needs and uses for data. 
Consequently, collecting data will less often be viewed as an externally imposed 
burden and cost. Whether providing these data to national surveys will be seen as 
burdensome, however, will depend critically on the ease with which data can be 
transmitted to those requesting information. To facilitate transmission, NCES will 
need to pay particular attention to assisting with the standardization of data ele- 
ments and with the development of cheap scannable forms and other strategies 
for promoting electronic access and transfer. 

Second, as local educators and administrators 
become more sophisticated users of data (rather 
than just providers), the business of designing sur- 
veys, collecting and analyzing data, and reporting 
results is likely to become much more interactive. 

The traditional model in which NCES assumes 
primary responsibility for all of these functions is 
likely to yield to much more decentralized, dis- 
tributed models in which the respective roles of 
surveyor and respondent become less distinct and 
more intertwined. For example, respondents who 
are also users of data may play a much greater role 
in defining survey questions and data elements. 

They may also develop specialized analyses 
(including analytic routines) that are shared with 
other respondent/ users. NCES, in turn, may 
assume more responsibility for coordinating and brokering surveys, analyses, and 
reporting, rather than unilaterally directing and conducting all of these activities. 
One possible implication of these trends is a reversal in the respective roles of 
independent surveys and administrative records in providing national data for 
education. To date, administrative records have mainly been adjuncts to large- 
scale surveys; they have supplemented data collected through written question- 
naires or telephone interviews. In the not-too-distant future, administrative 
records may become the basic building blocks of national data systems, with 
smaller targeted questionnaires designed as supplements. 
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Methods for Producing Better Statistical Estimates 



To provide good information for policy, research, and practice, NCES relies on a 
wide variety of survey design, data collection, and analytic methods — some rela- 
tively simple and widely known, others extraordinarily complex or reflecting 
recent advances in specialized fields. For purposes of this synthesis, a thorough 
discussion of survey and analytic methods is neither possible nor appropriate. 
The papers produced for this project, however, raised and discussed a number of 
methodological issues and developments. Four of these deserve special mention 
for careful consideration by NCES in charting its future course.^ 

First, as mentioned previously, NCES should 
exploit opportunities to combine well-designed, 
targeted controlled experiments within the nation- 
al surveys that have been the hallmark of its data 
collection activities. These experiments must be 
compatible with the mission and conduct of the 
larger survey effort, and a particular experiment 
should not be undertaken if it risks jeopardizing 
the nationally representative and descriptive 
power of the survey in which it is imbedded. 
Fiowever, if these criteria can be satisfied, imbed- 
ded experiments are promising examples of constructing a "whole exceeding the 
sum of its parts." Such experiments could contribute significantly to knowledge 
about what works and why in the nation's classrooms. 
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Second, survey questions that elicit information on sensitive topics must always 
be considered with great care. National surveys about education are no excep- 
tion, and it is important that NCES does not avoid issues simply because they are 
sensitive or controversial. Methodological developments can help reduce some of 
the concern surrounding this issue. For example, one promising strategy called 
"network-based estimating," in which respondents are asked about the behavior 
of unidentified acquaintances in their social network, has been developed by 
quantitative anthropologists. There is growing evidence that this procedure pro- 
duces indirect but reliable information on sensi- 
tive topics, without depending on the respondent 
to report directly on his or her own personal expe- 
rience. NCES should explore the feasibility of 
using this or similar techniques in future surveys. 

Third, and related to the second issue, there are 
promising new developments in methods for gen- 
erating indirect estimates of statistics at subnation- 
al levels or for intervening periods of time 
between surveys. Traditionally, producing esti- 
mates for smaller units of analysis — states, for 
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example, or institutions within states — has 
depended primarily on increasing sample size. 
Similarly, obtaining estimates more frequently — 
say, every 5 years rather than every 10 — typically 
requires administering the survey more frequently. 
Both of these strategies are usually quite expensive. 
An alternative method being developed uses auxil- 
iary data (from ongoing administrative records, for 
example) along with the survey data to produce 
indirect estimates for smaller units of analysis or 



I f NCES were able to 
significantly improve the 
connections between its 
surveys, it is likely that 
opportunities for better, 
more focused research 
would be enhanced. 



intervening periods of time. Successfully adapting these techniques to some NCES 
surveys could yield more finely grained estimates at a modest cost. 



Fourth, there are long-standing calls for better linking and integrating the data- 
bases produced by NCES surveys. As noted earlier, a better understanding of 
how learning occurs in the classroom will require simultaneous access to data on 
curriculum content, teaching practices, student behavior, and student outcomes. 
It has been rare to find a single database with rich information on all of these 
attributes for a sufficiently large sample, however. If NCES were able to signifi- 
cantly improve the connections between its surveys, it is likely that opportunities 
for better, more focused research would be enhanced. However, in order to 
achieve this long sought-after objective, NCES must do substantial work, both 
conceptually and methodologically, to determine precisely what is meant by 
"linking" and "integrating." 

Finally, NCES must continue to actively promote methodological developments 
and adaptations suited to its mission. Most researchers, whether engaged in a 
particular substantive pursuit or methodological advancement, are occupied pri- 
marily with their own interests and agendas; they are not paying much attention 
to the relevance of their work for NCES. Consequently, NCES needs to provide 
for the orderlv acquisition and screening of methodological and technological 
applications to surveys and analysis. There are many strategies for doing this, 
including advisory groups, grants, conferences, commissioned papers, and so on. 
Whatever strategy is chosen, however, the basic objective must be an explicit and 
high-priority item on the agenda of NCES. 



Internet and the World Wide Web 

No discussion of future technological developments would be complete without 
some mention of the Internet and the World Wide Web.^^ However, the pace and 
variety with which these are evolving make any effort to forecast precisely their 
role in the work of NCES quite difficult, if not simply foolish. Perhaps, the most 
useful approach is to use the evolution of the information highway as a metaphor 
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Percentage of Students Who Used a Computer at School or 
AT Home, by Family Income and Year, Grades 7-12: 
1984, 1989, AND 1993 
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SOURCE: U.S. Department of Education. National Center for Education 
Statistics, The Condition of Education 1995, Washington, D.C., 34-35. 



for changes in communication 
and interaction between NCES 
and the public it serves. In some 
respects, the Internet and the 
World Wide Web will facilitate 
and hasten these changes, but in 
others, they are simply reflective 
of larger forces at work in contem- 
porary society. 

Presently, NCES is in the second 
stage of a three-stage evolution in 
how many organizations typically 
interact with their clients. In Stage 
One, to accomplish its mission, 
NCES dominates the relationship 
between itself and those who 
either provide or use the data it 
gathers. Communication tends to 
be mainly one-way and follows 
well-established paths. NCES 
designs the surveys, administers 



questionnaires or interviews, collects and cleans data, conducts its own analysis, 
and produces and disseminates reports. Other analysts of NCES data pursue their 
research independently; they do not feed back results to NCES, at least in any 
systematic fashion. Stage One loosely represents the pre-Internet, pre-Web world. 

It is history. 

In Stage Two, which coincides with the advent and initial development of elec- 
tronic networks, relations between NCES and providers and users of data become 
more interactive (though still predominantly one-way), and in some instances the 
distinction between data provider and user begins to blur. In this stage, surveys 
begin to make more use of electronically stored administrative records, and, con- 
sequently, questions or data elements may be tailored to a particular respondent. 
Selected providers and users may be authorized limited on-line access data to 
update or correct information. Although NCES continues to generate substantial 
analyses on its own, it also begins to pay closer attention to the analytic objectives 
of users. In addition to distributing data files, it also disseminates analysis files 
designed to facilitate specific types of research — the relationship between educa- 
tion and labor market participation, for example. Additionally, NCES may pro- 
vide analysts with software to accelerate their analyses or to ensure that those 
who conduct external analyses of NCES data adopt appropriate statistical tech- 
niques. Reports are made available in electronic form, and specialized electronic 
user groups or technical review panels begin to form on the network. Currently, 
NCES is already well immersed in Stage Two. 
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In Stage Three, which will emerge more clearly 
and strongly with greater access to electrorvic net- 
works and with deeper understanding about how 
to use them effectively, relationships between 
NCES and its data providers and users will 
become truly two-way and continuous. Any data 
user, who could also be a data provider (a state 
office, for example), might send NCES a small soft- 
ware program that initiates a customized database 
search, adds the results to NCES's data library, and 
returns a tailored report to the original requester. 

Conversely, NCES may be constantly developing 
small software programs that go out over the net- 
work and retrieve data needed to respond to spe- 
cific inquiries from Congress, researchers, educa- 
tors, or the public at large.^ 

Surveys may assume the form of database devel- 
opment, with specifications designed interactively 
by users and providers coordinated by NCES. The scale of written question- 
naires or telephone interviews will diminish considerably or be limited to highly 
focused inquiries. Much of this design process will occur on-line through elec- 
tronic conferencing among NCES, data users, and data providers. Even though 
NCES will probably still produce many of its own reports, electronic versions 
of these documents will contain numerous electronic links to other data sets, 
technical references, and related reports. They may also contain interactive soft- 
ware that will permit users to perform "what if" analyses while perusing a 
report and to generate customized tables or graphics. Alternatively, users will 
generate their own electronic reports and analyses and create links to NCES 
documents residing in electronic networks. Precisely what is generated by 
NCES and what is generated by others may become less easy to distinguish. 

Stage Three is not here yet, and it will probably look quite different from this 
admittedly inchoate prediction. However, this stage will probably arrive much 
sooner than expected. The more NCES can anticipate and help shape these devel- 
opments, the more likely it will be able to use them effectively to report on the 
condition of education in the United States and other countries. 
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Qonclusion 

From data to information— transforming quantitative facts about education into 
knowledge useful to policymakers, researchers, practitioners, and the general 
public — this aim has always been central to the mission of the National Center for 
Education Statistics. In and of itself, this objective is not a new direction for the 
agency. However, what constitutes useful information and how it gets produced, 
distributed, and used are changing. To keep pace with these changes, indeed to 
stay out in front and help shape their development, NCES must chart some new 
directions. 



Probably the most fundamental change that NCES 
will need to address is its emergence as a provider 
of information services and systems, rather than a 
primary collector and provider of data per se. In 
today's climate of growing demands for informa- 
tion, but limited resources to produce it, NCES will 
need to pay particular attention to assuming new 
roles as a facilitator, broker, translator, linkage, fil- 
ter, and pathfinder in a complex web of providers 
and users of education data. To these new roles, 
the agency can bring a strong foundation of stan- 
dards for high-quality data and analysis, as well as a firm understanding of the 
kinds of information that are most relevant to deliberating national policy for 
education. 

As these new roles develop, NCES may find itself shedding or at least de-empha- 
sizing old functions. Data collection that occurs independently of front-line 
administrative and teaching systems and their own information needs is likely to 
diminish significantly. This change, in combination with technological advances, 
may lead to data collection systems that are far more decentralized, interactiv^e, 
and operating in "real time" than the systems that have traditionally supported 

national surveys. It is ev^en possible that eventual- 
ly NCES may find that it is no longer in the data 
collection business, as this function has traditional- 
ly been defined. Instead, it will be primarily a sys- 
tems manager and analyst, a producer and broker 
of information for ongoing nationally oriented 
assessments, as well as thousands of state and local 
customized queries. Data collection and storage, 
however, may occur largely outside of the imme- 
diate domain of NCES. 
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"Reporting statistics and information showing the condition and progress of edu- 
cation in the United States and other nations in order to promote and accelerate 
the improvement of American education" — this charge is a lasting mission for the 
National Center for Education Statistics. Fulfilling it successfully will require 
careful attention to changing national priorities, a strong commitment to improv- 
ing education research and practice, and an openness to recognizing and adopt- 
ing important advances in methods and technology. 
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NOTES 



1. U.S. Department of Education, National Center for Education Statistics, "Common Core of 
Data" and "Financial Statistics of Institutions of Higher Education," surveys and unpublished 
data, FY 94-95. 

2. Section 402(b) of the National Education Statistics Act of 1994 (20 U.S.C. 9001). 

3. Indeed, there are now standards for all of NCES major activities — survey planning and testing, 
statistical processing, data provision and analysis, evaluation and documentation, and contract 
management and operations. See U.S. Department of Education. 1992. NCES Statistical 
Standards. Washington, D.C.: National Center for Education Statistics. 

4. These follow the principles for a federal statistical agency developed in Martin and Straf, eds. 
(1992). 

5. For a full description of the current program of work at NCES, see U.S. Department of 
Education. 1995a. Programs and Plans: 1995 Edition. Washington, D.C.: National Center for 
Education Statistics. 

6. SASS also surveys private school teachers in a sample of schools drawn from the Private School 
Universe file maintained by NCES. 

7. At present, legislation requires that NAEP assess reading and mathematics ever\' 2 years; sci- 
ence and writing at least every 4 years; and history, geography, and other subjects determined 
by the National Assessment Governing Board at least every 6 years. 

8. In particular, see the following papers in From Data to Information: Nezo Directions for the 
Natiojial Center for Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. 
Department of Education, National Center for Education Statistics, 1996): D. Brewer and C. 
Stasz, "Enhancing Opportunity to Learn Measures in NCES Data," pp. 3-1-3-28; D. Mandel, 
"Teacher Education, Training, and Staff Development: Implications for National Surveys," pp. 
3-29-3-42; M. McPherson and M. Schapiro, "Tracking the Costs and Benefits of Postsecondary 
Education: Implications for National Surveys," pp. 6-1-6-12; D. Breneman and F. Galloway, 
"Special Issues in Postsecondary Education and Lifelong Learning," pp. 6-13-6-28; P. Cappelli, 
"Education and Work: Curriculum, Performance, and Job-Related Outcomes," pp. 8-1-8-34. 

9. See especially the following papers in From Data to Information: Nezo Directions for the National 
Center for Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of 
Education, National Center for Education Statistics, 1996): J. Jennings and D. Stark, "Tracking 
Education Reform: What Type of National Data Should Be Collected Through 2010?" pp. 2- 
1-2-11; C. Cross and A. Stempel, "Where Are We Going? Policy Implications for Data 
Collection Through 2010," pp. 2-12-2-18. 

10. See the following papers in Froui Data to Information: Nezo Directions for the National Center for 
Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of Education, 
National Center for Education Statistics, 1996): C. Metcalf, "Incorporating Experimental 
Designs Into New NCES Data Collection Methodologies," pp. 5-1-5-18; R. Boruch and G. 
Terhanian, "'So What?' The Implications of New Analytic Methods for Designing NCES 
Surveys," pp. 4-1-4-115. 

11. See the discussion in Boruch and Terhanian, "'So What?'." These authors also suggest that 
NCES consider adopting a "satellite" policy that would permit including controlled experi- 
mental studies in national surveys in a fashion similar to the way NASA allows adjuncts to 
space missions for astrophysicists and others. 

12. See Brewer and Stasz, "Enhancing Opportunities," and Mandel, "Teacher Education." 
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13. For a summary of trends in technological capacity to store, retrieve, and analyze data, see G. 
Ligon, "New Developments in Technology: Implications for Collecting, Storing, Retrieving, 
and Disseminating National Data for Education," pp. 9-32-9-*65 in From Data to Information: 
New Directions for the National Center for Education Statistics, ed. Gary Hoachlander 
(Washington, D.C.: U.S. Department of Education, National Center for Education Statistics, 
1996). 

14. See J. Stigler, "Large-Scale Video Surv^eys for the Study of Classroom Processes," pp. 7-1-7-29 
in From Data to Information: New Directions for the National Center for Education Statistics, ed. Gary 
Hoachlander (Washington, D.C.: U.S. Department of Education, National Center for Education 
Statistics, 1996). 

15. See McPherson and Schapiro, "Tracking the Costs." 

16. While SASS provides additional data on salaries of administrative and instructional personnel 
at the elementary and secondary level, NPSAS provides additional postsecondary information 
on tuition and costs. 

17. See Cappelli, "Education and Work." 

18. There are important confidentiality considerations that must be addressed if this kind of ser- 
vice were to be provided. However, with explicit attention to benchmarking at the outset of a 
survey, problems surrounding confidentiality could be reduced. 

19. See Mandel, "Teacher Education." 

20. Video already plays an important role in the certification process used by NBPTS, and NCES 
could build on the experience of the Board, as well as that of other researchers developing this 
technology. 

21. See the following papers in From Data to Information: New Directions for the National Center for 
Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of Education, 
National Center for Education Statistics, 1996): G. Ligon, "New Developments in Technology: 
Implications for Collecting, Storing, Retrieving, and Disseminating National Data for 
Education," pp. 9-32-9-65; F. Scheuren, "Administrative Record Opportunities in Education 
Survey Research," pp. 9-1-9-31. 

22. Electronic recordkeeping is still far from universal, especially at the elementary and secondary 
levels; paper files are still the norm in many places. 

23. These are developed in more detail in the papers by Boruch and Terhanian, "'So What?'," 
Metcalf, "Incorporating Experimental Designs," and Scheuren, "Administrative Record 
Opportunities." 

24. For more information about specific opportunities for NCES to use electronic networks, see the 
papers by Boruch and Terhanian, "'So What?'," Ligon, "New Developments," and Scheuren, 
"Administrative Record Opportunities." 

25. The continuing development of "object technology," a technique for more rapidly constructing 
software programs out of many small object modules, should hasten the explosion of this sort 
of interactive dissemination and sharing of programs. 
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ROCEEDINGS 



From Data to Information: New Directions for the National 
Center for Education Statistics 



The conference was held in Washington, DC on November 27-29, 1995. For the 
complete text of the proceedings, please see NCES Publication 96-901, From Data 
to Information: New Directions for the National Center for Education Statistics, which 
can be ordered from NCES Project Officer Edith McArthur at (202) 219-1442. 

1 From Data to Information: New Directions for the National Center for 

Education Statistics 

Gary Hoachlander: From Data to Information: New Directions for the National 
Center for Education Statistics 

Emerson Elliott: Introductory Comments 

2 Tracking Education Reform: Implications for Collecting National Data 

Through 2010 

John F. Jennings and Diane Stark: Tracking Education Reform: What Type of 
National Data Should Be Collected Through 2010? 

Christopher T. Cross and Amy Rukea Stempel: Where Are We Going? Policy 
Implications for Data Collection Through 2010 

Discussant Comments by Mary J. Frase 

3 Curriculum, Pedagogy, and Professional Development 

Dominic J. Brewer and Cathleen Stasz: Enhancing Opportunity to Learn 
Measures in NCES Data 

David R. Mandel: Teacher Education, Training, and Staff Development: 
Implications for National Surveys 

Discussant Comments by Michael Timpane, Eileen M. Sclan, Mary Rollefson, 
and Sharon Bobbitt 
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4 Trends in Statistical and Analytic Methodology: Implications for National 
Surveys 

Robert F. Boruch and George Terhanian: "So What?" The Implications of New 
Analytic Methods for Designing NCES Surveys 
Discussant Comments by Frederick Mosteller 
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5 New Data Collection Methodologies, Part II: Experimental Design 

Charles E. Metcalf: Incorporating Experimental Designs Into Neiu NCES Data 
Collection Methodologies 

Discussant Comments by Donald R. Rubin 

6 Postsecondary Education 

Michael S. McPherson and Morton O. Schapiro: Tracking the Costs and Benefits 
of Postsecondan/ Education: Implications for National Surveys 

David W. Breneman and Frederick J. Galloway: Special Issues in Postsecondary 
Education and Lifelong Learning 

Discussant Comments by Jamie Merisotis, Jim McKenney, and Paula Knepper 
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James W. Stigler: Large-Scale Video Surveys for the Study of Classroom Processes 
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Outcomes 

Discussant Comments by David Stern 

9 Using Administrative Records and New Developments in Technology 
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Glynn D. Ligon: New Developments in Technology: Implications for Collecting, 
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